Neural Q-learning for discrete-time nonlinear zero-sum games with adjustable convergence rate.

In this paper, an adjustable Q-learning scheme is developed to solve the discrete-time nonlinear zero-sum game problem, which can accelerate the convergence rate of the iterative Q-function sequence. First, the monotonicity and convergence of the iterative Q-function sequence are analyzed under some conditions. Moreover, by employing neural networks, the model-free tracking control problem can be overcome for zero-sum games. Second, two practical algorithms are designed to guarantee the convergence with accelerated learning. In one algorithm, an adjustable acceleration phase is added to the iteration process of Q-learning, which can be adaptively terminated with convergence guarantee. In another algorithm, a novel acceleration function is developed, which can adjust the relaxation factor to ensure the convergence. Finally, through a simulation example with the practical physical background, the fantastic performance of the developed algorithm is demonstrated with neural networks.

Diversified Knowledge Transfer Strategy for Multitasking Particle Swarm Optimization.

Evolutionary multitasking optimization (EMTO) has capability of performing a population of individuals together by sharing their intrinsic knowledge. However, the existed methods of EMTO mainly focus on improving its convergence using parallelism knowledge belonging to different tasks. This fact may lead to the problem of local optimization in EMTO due to unexploited knowledge on behalf of the diversity. To address this problem, in this article, a diversified knowledge transfer strategy is proposed for multitasking particle swarm optimization algorithm (DKT-MTPSO). First, according to the state of population evolution, an adaptive task selection mechanism is introduced to manage the source tasks that contribute to the target tasks. Second, a diversified knowledge reasoning strategy is designed to capture the knowledge of convergence, as well as the knowledge associated with diversity. Third, a diversified knowledge transfer method is developed to expand the region of generated solutions guided by acquired knowledge with different transfer patterns so that the search space of tasks can be explored comprehensively, which is favor of EMTO alleviating local optimization. Finally, the performance of the proposed algorithm is evaluated in comparison with some other state-of-the-art EMTO algorithms on multiobjective multitasking benchmark test suits, and the practicality of the algorithm is verified in a real-world application study. The results of experiments demonstrate the superiority of DKT-MTPSO compared to other algorithms.

Self-Organizing Robust Fuzzy Neural Network for Nonlinear System Modeling.

Fuzzy neural network (FNN) is a structured learning technique that has been successfully adopted in nonlinear system modeling. However, since there exist uncertain external disturbances arising from mismatched model errors, sensor noises, or unknown environments, FNN generally fails to achieve the desirable performance of modeling results. To overcome this problem, a self-organization robust FNN (SOR-FNN) is developed in this article. First, an information integration mechanism (IIM), consisting of partition information and individual information, is introduced to dynamically adjust the structure of SOR-FNN. The proposed mechanism can make itself adapt to uncertain environments. Second, a dynamic learning algorithm based on the α -divergence loss function ( α -DLA) is designed to update the parameters of SOR-FNN. Then, this learning algorithm is able to reduce the sensibility of disturbances and improve the robustness of Third, the convergence of SOR-FNN is given by the Lyapunov theorem. Then, the theoretical analysis can ensure the successful application of SOR-FNN. Finally, the proposed SOR-FNN is tested on several benchmark datasets and a practical application to validate its merits. The experimental results indicate that the proposed SOR-FNN can obtain superior performance in terms of model accuracy and robustness.

Application of TCN-biGRU neural network in [Formula: see text] concentration prediction.

Fine particulate matter ([Formula: see text]) poses a significant threat to human life and health, and therefore, accurately predicting [Formula: see text] concentration is critical for controlling air pollution. Two improved types of recurrent neural networks (RNNs), the long short-term memory (LSTM) and gated recurrent unit (GRU), have been widely used in time series data prediction due to their ability to capture temporal features. However, both degrade into random guessing as the time length increases. In order to enhance the accuracy of [Formula: see text] concentration prediction and address the issue of random guessing in RNNs neural networks, this study introduces a TCN-biGRU neural network model. This model is a hybrid prediction approach based on combining temporal convolutional networks (TCN) and bidirectional gated recurrent units (bi-GRU). TCN extracts higher-level feature information from longer time series data of [Formula: see text] concentrations, while bi-GRU captures features from past and future data to achieve more accurate predictive outcomes. This case study utilizes data from monitoring stations in Beijing in 2021 for conducting [Formula: see text] prediction experiments. The TCN-biGRU model achieves an average absolute error, root mean square error, and [Formula: see text] of 4.20, 7.71, and 0.961 in its predictive outcomes. When compared to the predictive outcomes of individual LSTM, GRU, and bi-GRU models, it is evident that the TCN-biGRU model exhibits smaller errors and superior predictive performance.

Intelligent-Critic-Based Tracking Control of Discrete-Time Input-Affine Systems and Approximation Error Analysis With Application Verification.

In recent years, the application of function approximators, such as neural networks and polynomials, has ushered in a new stage of development in solving optimal control problems. However, considering the existence of approximation errors, the stability of the controlled system cannot be guaranteed. Therefore, in view of the prevalence of approximation errors, we investigate optimal tracking control problems for discrete-time systems. First, a novel value function is introduced into the intelligent critic framework. Second, an implicit method is utilized to demonstrate the boundedness of the iterative value functions with approximation errors. An explicit method is applied to prove the stability of the system with approximation errors. Furthermore, an evolving policy is designed to iteratively tackle the optimal tracking control problem and demonstrate the stability of the system. Finally, the effectiveness of the developed method is verified through numerical as well as practical examples.

Anammox granule destruction and reconstruction in a partial nitrification/anammox system under hydroxylamine stress.

Nitrite oxidizing bacteria (NOB) outcompeting anammox bacteria (AnAOB) poses a challenge to the practical implementation of the partial nitrification/anammox (PN/A) process for municipal wastewater. A granules-based PN/A bioreactor was operated for 260 d with hydroxylamine (NH2OH) added halfway through. qPCR results detected the different amounts of NOB among granules and flocs and the dynamic succession during operation. CLSM images revealed a unique layered structure of granules that NOB located inside led to the inhibition effect of NH2OH delayed. Besides, the physical and morphological characteristics revealed that anammox granules experienced destruction. AnAOB took the broken granules as an initial biofilm aggregate to reconstruct new granules. RT-qPCR and high throughput sequencing results suggested that functional gene expression and community structure were regulated for the AnAOB metabolism process. Correspondingly, the rapid proliferation (0.52 â†’ 1.99%) of AnAOB was realized, and the nitrogen removal rate achieved a nearly quadruple improvement (0.21 â†’ 0.83 kg-N/m3·d). This study revealed that anammox granules can self-reconstruct in the PN/A system when granules are disintegrated under NH2OH stress, broadening the feasibility of applying PN/A process.

Dichotomy value iteration with parallel learning design towards discrete-time zero-sum games.

In this paper, a novel parallel learning framework is developed to solve zero-sum games for discrete-time nonlinear systems. Briefly, the purpose of this study is to determine a tentative function according to the prior knowledge of the value iteration (VI) algorithm. The learning process of the parallel controllers can be guided by the tentative function. That is to say, the neighborhood of the optimal cost function can be compressed within a small range via two typical exploration policies. Based on the parallel learning framework, a novel dichotomy VI algorithm is established to accelerate the learning speed. It is shown that the parallel controllers will converge to the optimal policy from contrary initial policies. Finally, two typical systems are used to demonstrate the learning performance of the constructed dichotomy VI algorithm.

Fuzzy Boundary Sampled-Data Control for Nonlinear Parabolic DPSs.

For a nonlinear parabolic distributed parameter system (DPS), a fuzzy boundary sampled-data (SD) control method is introduced in this article, where distributed SD measurement and boundary SD measurement are respected. Initially, this nonlinear parabolic DPS is represented precisely by a Takagi-Sugeno (T-S) fuzzy parabolic partial differential equation (PDE) model. Subsequently, under distributed SD measurement and boundary SD measurement, a fuzzy boundary SD control design is obtained via linear matrix inequalities (LMIs) on the basis of the T-S fuzzy parabolic PDE model to guarantee exponential stability for closed-loop parabolic DPS by using inequality techniques and a LF. Furthermore, respecting the property of membership functions, we present some LMI-based fuzzy boundary SD control design conditions. Finally, the effectiveness of the designed fuzzy boundary SD controller is demonstrated via two simulation examples.

Adaptive event-triggered extended dissipative synchronization of delayed reaction-diffusion neural networks under deception attacks.

Under spatially averaged measurements (SAMs) and deception attacks, this article mainly studies the problem of extended dissipativity output synchronization of delayed reaction-diffusion neural networks via an adaptive event-triggered sampled-data (AETSD) control strategy. Compared with the existing ETSD control methods with constant thresholds, our scheme can be adaptively adjusted according to the current sampling and latest transmitted signals and is realized based on limited sensors and actuators. Firstly, an AETSD control scheme is proposed to save the limited transmission channel. Secondly, some synchronization criteria under SAMs and deception attacks are established by utilizing Lyapunov-Krasovskii functional and inequality techniques. Then, by solving linear matrix inequalities (LMIs), we obtain the desired AETSD controller, which can satisfy the specified level of extended dissipativity behaviors. Lastly, one numerical example is given to demonstrate the validity of the proposed method.

LASSO and attention-TCN: a concurrent method for indoor particulate matter prediction.

Long time exposure to indoor air pollution environments can increase the risk of cardiovascular and respiratory system damage. Most previous studies focus on outdoor air quality, while few studies on indoor air quality. Current neural network-based methods for indoor air quality prediction ignore the optimization of input variables, process input features serially, and still suffer from loss of information during model training, which may lead to the problems of memory-intensive, time-consuming and low-precision. We present a novel concurrent indoor PM prediction model based on the fusion model of Least Absolute Shrinkage and Selection Operator (LASSO) and an Attention Temporal Convolutional Network (ATCN), together called LATCN. First, a LASSO regression algorithm is used to select features from PM1, PM2.5, PM10 and PM (>10) datasets and environmental factors to optimize the inputs for indoor PM prediction model. Then an Attention Mechanism (AM) is applied to reduce the redundant temporal information to extract key features in inputs. Finally, a TCN is used to forecast indoor particulate concentration in parallel with inputting the extracted features, and it reduces information loss by residual connections. The results show that the main environmental factors affecting indoor PM concentration are the indoor heat index, indoor wind chill, wet bulb temperature and relative humidity. Comparing with Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) approaches, LATCN systematically reduced the prediction error rate (19.7% ~ 28.1% for the NAE, and 16.4% ~ 21.5% for the RMSE) and improved the model running speed (30.4% ~ 81.2%) over these classical sequence prediction models. Our study can inform the active prevention of indoor air pollution, and provides a theoretical basis for indoor environmental standards, while laying the foundations for developing novel air pollution prevention equipment in the future.

Convergence and Stability of Optimal Regulation via Generalized N -Step Value Gradient Learning.

In this article, the generalized N -step value gradient learning (GNSVGL) algorithm, which takes a long-term prediction parameter λ into account, is developed for infinite horizon discounted near-optimal control of discrete-time nonlinear systems. The proposed GNSVGL algorithm can accelerate the learning process of adaptive dynamic programming (ADP) and has a better performance by learning from more than one future reward. Compared with the traditional N -step value gradient learning (NSVGL) algorithm with zero initial functions, the proposed GNSVGL algorithm is initialized with positive definite functions. Considering different initial cost functions, the convergence analysis of the value-iteration-based algorithm is provided. The stability condition for the iterative control policy is established to determine the value of the iteration index, under which the control law can make the system asymptotically stable. Under such a condition, if the system is asymptotically stable at the current iteration, then the iterative control laws after this step are guaranteed to be stabilizing. Two critic neural networks and one action network are constructed to approximate the one-return costate function, the λ -return costate function, and the control law, respectively. It is emphasized that one-return and λ -return critic networks are combined to train the action neural network. Finally, via conducting simulation studies and comparisons, the superiority of the developed algorithm is confirmed.

Neural Network-Based Adaptive Tracking Control for Denitrification and Aeration Processes With Time Delays.

Wastewater treatment process (WWTP), consisting of a class of physical, chemical, and biological phenomena, is an important means to reduce environmental pollution and improve recycling efficiency of water resources. Considering characteristics of the complexities, uncertainties, nonlinearities, and multitime delays in WWTPs, an adaptive neural controller is presented to achieve the satisfying control performance for WWTPs. With the advantages of radial basis function neural networks (RBF NNs), the unknown dynamics in WWTPs are identified. Based on the mechanistic analysis, the time-varying delayed models of the denitrification and aeration processes are established. Based on the established delayed models, the Lyapunov-Krasovskii functional (LKF) is used to compensate for the time-varying delays caused by the push-flow and recycle flow phenomenon. The barrier Lyapunov function (BLF) is used to ensure that the dissolved oxygen (DO) and nitrate concentrations are always kept within the specified ranges though the time-varying delays and disturbances exist. Using Lyapunov theorem, the stability of the closed-loop system is proven. Finally, the proposed control method is carried out on the benchmark simulation model 1 (BSM1) to verify the effectiveness and practicability.

Double-Closed-Loop Robust Optimal Control for Uncertain Nonlinear Systems.

Optimal control methods have gained significant attention due to their promising performance in nonlinear systems. In general, an optimal control method is regarded as an optimization process for solving the optimal control laws. However, for uncertain nonlinear systems with complex optimization objectives, the solving of optimal reference trajectories is difficult and significant that might be ignored to obtain robust performance. For this problem, a double-closed-loop robust optimal control (DCL-ROC) is proposed to maintain the optimal control reliability of uncertain nonlinear systems. First, a double-closed-loop scheme is established to divide the optimal control process into a closed-loop optimization process that solves optimal reference trajectories and a closed-loop control process that solves optimal control laws. Then, the ability of the optimal control method can be improved to solve complex uncertain optimization problems. Second, a closed-loop robust optimization (CL-RO) algorithm is developed to express uncertain optimization objectives as data-driven forms and adjust optimal reference trajectories in a close loop. Then, the optimality of reference trajectories can be improved under uncertainties. Third, the optimal reference trajectories are tracked by an adaptive controller to derive the optimal control laws without certain system dynamics. Then, the adaptivity and reliability of optimal control laws can be improved. The experimental results demonstrate that the proposed method can achieve better performance than other optimal control methods.

Deterministic Learning-Based Adaptive Neural Control for Nonlinear Full-State Constrained Systems.

In this article, an adaptive neural learning method is introduced for a category of nonlinear strict-feedback systems with time-varying full-state constraints. The two challenging problems of state constraints and learning capability are investigated and solved in a unified framework. To obtain the learning of unknown functions and satisfy full-state constraints, three main steps are considered. First, an adaptive dynamic surface controller (DSC) based on barrier Lyapunov functions (BLFs) is structured to implement that the closed-loop systems signals are bounded and full-state variables remain within the prescribed time-varying intervals. Moreover, the radial basis function neural networks (RBF NNs) are used to identify unknown functions. The output of the first-order filter, instead of virtual control derivatives, is used to simplify the complexity of the RBF NN input variables. Second, the state transformation is used to obtain a class of linear time-varying subsystems with small perturbations such that the recurrence of the RBF NN input variables and the partial persistent excitation condition are actualized. Therefore, the unknown functions can be accurately approximated, and the learned knowledge is kept as constant NN weights. Third, the obtained constant weights are borrowed into an adaptive learning scheme to achieve the batter control performance. Finally, simulation studies illustrate the advantage of the reported adaptive learning method on higher tracking accuracy, faster convergence rate, and lower computational expense by reusing learned knowledge.

Transfer Learning Algorithm With Knowledge Division Level.

One of the major challenges of transfer learning algorithms is the domain drifting problem where the knowledge of source scene is inappropriate for the task of target scene. To solve this problem, a transfer learning algorithm with knowledge division level (KDTL) is proposed to subdivide knowledge of source scene and leverage them with different drifting degrees. The main properties of KDTL are three folds. First, a comparative evaluation mechanism is developed to detect and subdivide the knowledge into three kinds-the ineffective knowledge, the usable knowledge, and the efficient knowledge. Then, the ineffective and usable knowledge can be found to avoid the negative transfer problem. Second, an integrated framework is designed to prune the ineffective knowledge in the elastic layer, reconstruct the usable knowledge in the refined layer, and learn the efficient knowledge in the leveraged layer. Then, the efficient knowledge can be acquired to improve the learning performance. Third, the theoretical analysis of the proposed KDTL is analyzed in different phases. Then, the convergence property, error bound, and computational complexity of KDTL are provided for the successful applications. Finally, the proposed KDTL is tested by several benchmark problems and some real problems. The experimental results demonstrate that this proposed KDTL can achieve significant improvement over some state-of-the-art algorithms.

Stability and Admissibility Analysis for Zero-Sum Games Under General Value Iteration Formulation.

In this article, the general value iteration (GVI) algorithm for discrete-time zero-sum games is investigated. The theoretical analysis focuses on stability properties of the systems and also the admissibility properties of the iterative policy pair. A new criterion is established to determine the admissibility of the current policy pair. Besides, based on the admissibility criterion, the improved GVI algorithm toward zero-sum games is developed to guarantee that all iterative policy pairs are admissible if the current policy pair satisfies the criterion. On the basis of the attraction domain, we demonstrate that the state trajectory will stay in the region using the fixed or the evolving policy pair if the initial state belongs to the domain. It is emphasized that the evolving policy pair can stabilize the controlled system. These theoretical results are applied to linear and nonlinear systems via offline and online critic control design.

Adaptive Critic for Event-Triggered Unknown Nonlinear Optimal Tracking Design With Wastewater Treatment Applications.

In this article, an event-based near-optimal tracking control algorithm is developed for a class of nonaffine systems. First, in order to gain the tracking control strategy, the costate function is established through the iterative dual heuristic dynamic programming (DHP) algorithm. Then, the event-based control method is employed to improve the utilization efficiency of resources and ensure that the closed-loop system has an excellent control performance. Meanwhile, the input-to-state stability (ISS) is proven for the event-based tracking plant. In addition, three kinds of neural networks are used in the event-based DHP algorithm, which aims to identify the nonaffine nonlinear system, estimate the costate function, and approximate the tracking control law. Finally, a numerical experimental simulation is conducted to verify the effectiveness of the proposed scheme. Moreover, in order to further validate the feasibility, the algorithm is applied to the wastewater treatment plant to effectively control the concentrations of dissolved oxygen and nitrate nitrogen.

Self-Organizing Interval Type-2 Fuzzy Neural Network Using Information Aggregation Method.

Interval type-2 fuzzy neural networks (IT2FNNs) usually stack adequate fuzzy rules to identify nonlinear systems with high-dimensional inputs, which may result in an explosion of fuzzy rules. To cope with this problem, a self-organizing IT2FNN, based on the information aggregation method (IA-SOIT2FNN), is developed to avoid the explosion of fuzzy rules in this article. First, a relation-aware strategy is proposed to construct rotatable type-2 fuzzy rules (RT2FRs). This strategy uses the individual RT2FR, instead of multiple standard fuzzy rules, to interpret interactive features of high-dimensional inputs. Second, a comprehensive information evaluation mechanism, associated with the interval information and rotation information of RT2FR, is developed to direct the structural adjustment of IA-SOIT2FNN. This mechanism can achieve a compact structure of IA-SOIT2FNN by growing and pruning RT2FRs. Third, a multicriteria-based optimization algorithm is designed to optimize the parameters of IA-SOIT2FNN. The algorithm can simultaneously update the rotatable parameters and the conventional parameters of RT2FR, and further maintain the accuracy of IA-SOIT2FNN. Finally, the experiments showcase that the proposed IA-SOIT2FNN can compete with the state-of-the-art approaches in terms of identification performance.

System Stability of Learning-Based Linear Optimal Control With General Discounted Value Iteration.

For discounted optimal regulation design, the stability of the controlled system is affected by the discount factor. If an inappropriate discount factor is employed, the optimal control policy might be unstabilizing. Therefore, in this article, the effect of the discount factor on the stabilization of control strategies is discussed. We develop the system stability criterion and the selection rules of the discount factor with respect to the linear quadratic regulator problem under the general discounted value iteration algorithm. Based on the monotonicity of the value function sequence, the method to judge the stability of the controlled system is established during the iteration process. In addition, once some stability conditions are satisfied at a certain iteration step, all control policies after this iteration step are stabilizing. Furthermore, combined with the undiscounted optimal control problem, the practical rule of how to select an appropriate discount factor is constructed. Finally, several simulation examples with physical backgrounds are conducted to demonstrate the present theoretical results.

Resilient Output Synchronization of Heterogeneous Multiagent Systems With DoS Attacks Under Distributed Event-/Self-Triggered Control.

This article investigates the resilient output synchronization problem of a class of linear heterogeneous multiagent systems subjected to denial-of-service (DoS) attacks. Two types of control mechanisms, namely, event- and self-triggered control mechanisms, are presented so as to cut down unnecessary information transmission. Both of these two mechanisms are distributed, and thus, only local information of each agent and its neighboring agents is adopted for the event condition design. The DoS attacks are considered to be aperiodic, and the quantitative relationship between the attributes of the DoS attacks and the synchronization is also revealed. It is shown that the output synchronization can be achieved exponentially in the presence of DoS attacks under the proposed control mechanisms. The validness of the provided mechanisms is certified by a simulation example.